Full Log Collection

APO-OneAgent enables fault logs and full log collection by default. This document is used to configure the full log collection for APO-OneAgent. The product is based on ilogtail for secondary development to collect logs generated by applications.

Full Log Collection Configuration in Kubernetes Environment

Users who have followed the 《APO OneAgent #Kubernetes》 can use this section to configure full log collection.

1. Modify the ConfigMap named apo-ilogtail-user-config and add a new configuration file <your-file-name>.yaml.
2. Refer to the configuration template in this section and save the content to <your-file-name>.yaml.
3. Restart OneAgent.

1. Kubernetes Standard Output Collection Configuration Template

OneAgent enables Kubernetes standard output collection by default. If there are no additional requirements, there is no need to manually configure this template.

Container standard output uses the service_docker_stdout plugin. Reference Documentation

enable: true
inputs:
    # Log collection plugin, adjust as needed
  - Type: service_docker_stdout
    # k8s label filtering
    ExcludeK8sLabel:
      clickhouse.altinity.com/namespace: apo
      app.kubernetes.io/instance: apo
    # Multi-line log merging
    BeginLineCheckLength: 10
    BeginLineRegex: "\\d+.*"
processors:
  - Type: processor_wait_for_signal
    DisableSignalSampler: true
    ContentsRename:
      "__tag__:_container_id_": "_container_id_"
      "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_http
    # Server-side vector to receive logs, replace with the actual server address
    RemoteURL: http://apo-vector-svc:4310
    Convert:
      Protocol: custom_single
      Encoding: json

2. Kubernetes Container Log File Collection Configuration Template

Collecting text logs uses the file_log plugin. Reference Documentation

Note

1. When collecting log files within containers, it does not support collecting log files mounted on PVs.
2. The maximum depth of the monitored log directory should not be too large, otherwise it may exceed the number of files monitored by ilogtail, leading to collection failure. You can add different configuration file pipelines for different collection paths.

enable: true
inputs:
    # Text log collection plugin, adjust as needed
  - Type: file_log
    # Log directory path
    LogPath: /var/log/nginx/
    # Maximum depth of the monitored log directory, 0 means monitoring files in the current directory
    MaxDepth: 0
    # Log file name pattern
    FilePattern: "*.log"
    # Set to true for monitoring log files within containers
    ContainerFile: true
processors:
  - Type: processor_wait_for_signal
    DisableSignalSampler: true
    ContentsRename:
      "__tag__:_container_id_": "_container_id_"
      "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_http
    # Server-side vector to receive logs, replace with the actual server address
    RemoteURL: http://apo-vector-svc:4310
    Convert:
      Protocol: custom_single
      Encoding: json

Full Log Collection Configuration in Docker Environment

For users who have installed OneAgent using the 《APO OneAgent #Traditional》, configure full log collection based on the monitored service type using the configurations in this section.

1. In the OneAgent ilogtail configuration directory apo-one-agent-compose/config/apo-ilogtail, add a new configuration file <your-file-name>.yaml
2. Refer to the configuration template in this section and save the content to <your-file-name>.yaml
3. Restart OneAgent

1. Docker Standard Output Collection Configuration Template

Suitable for collecting standard output from Docker containers on traditional servers. Use the service_docker_stdout plugin for container standard output, refer to the documentation

enable: true
inputs:
    # Log collection plugin, adjust as needed
  - Type: service_docker_stdout
    # Multi-line log merging
    BeginLineCheckLength: 10
    BeginLineRegex: "\\d+.*"
processors:
  - Type: processor_wait_for_signal
    DisableSignalSampler: true
    ContentsRename:
      "__tag__:_container_id_": "_container_id_"
      "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_http
    # Server-side vector to receive logs, replace with the actual server address
    RemoteURL: http://apo-vector-svc:4310
    Convert:
      Protocol: custom_single
      Encoding: json

2. Docker Container Log File Collection Configuration Template

Use the file_log plugin for collecting text logs, refer to the documentation

Note

It is advisable not to set the maximum depth of the monitored log directory too large, as it may exceed the number of files that ilogtail can monitor, leading to collection failures. You can add different configuration file pipelines for different collection paths.

enable: true
inputs:
    # Text log collection plugin, adjust as needed
  - Type: file_log
    # Log directory path
    LogPath: /var/log/nginx/
    # Maximum depth of the monitored log directory, 0 means monitoring files in the current directory
    MaxDepth: 0
    # Log file name pattern
    FilePattern: "*.log"
    # Set to true to monitor log files within containers
    ContainerFile: true
processors:
  - Type: processor_wait_for_signal
    DisableSignalSampler: true
    ContentsRename:
      "__tag__:_container_id_": "_container_id_"
      "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_http
    # Server-side vector receiving logs, replace with the actual server address
    RemoteURL: http://apo-vector-svc:4310
    Convert:
      Protocol: custom_single
      Encoding: json

Full Log Collection Configuration in Traditional Server Environments

For users who have installed OneAgent using the 《APO OneAgent #Traditional》, configure full log collection based on the monitored service type using the configurations in this section.

1. In the OneAgent ilogtail configuration directory apo-one-agent-compose/config/apo-ilogtail, add a new configuration file <your-file-name>.yaml
2. Refer to the configuration template in this section and save the content to <your-file-name>.yaml
3. Restart OneAgent

1. Configuration Template for Collecting Log Files from Host Paths

Applicable to traditional services that save log files to host paths. Use the file_log plugin for text log collection, refer to the documentation

Note

It is advisable not to set the maximum depth of the monitored log directory too large, as it may exceed the number of files that ilogtail can monitor, leading to collection failures. You can add different configuration file pipelines for different collection paths.

enable: true
inputs:
  # Text log collection plugin, adjust as needed
- Type: file_log
  # Path to the log file within the container where the ilogtail process resides
  # By default, the host directory is mounted to /logtail_host within the container
  # Therefore, prepend /logtail_host to the host path
  LogPath: /logtail_host/home/
  # File name pattern for collecting text logs, supports wildcards
  # Supports multi-level directory matching, meaning all files matching the pattern in the specified directory (including all subdirectories) will be found.
  # The number of levels to search can be controlled via MaxDepth
  FilePattern: "*.log"
  # Maximum depth of the directory to be monitored
  # For example, with a maximum depth of 2, even if all files match the filePattern
  # Only logs from $LogPath/xx/xx/test.log will be collected
  # Logs from $LogPath/xx/xx/xx/test.log will not be collected
  MaxDepth: 4
processors:
  # Associate with the application's pid
- Type: processor_path_to_pid 
- Type: processor_wait_for_signal
  DisableSignalSampler: true
  ContentsRename:
    "__tag__:_container_id_": "_container_id_"
    "__tag__:__path__": "_source_"
flushers:
  - Type: flusher_http
    # Server-side vector to receive logs, replace with the actual server address
    RemoteURL: http://apo-vector-svc:4310
    Convert:
      Protocol: custom_single
      Encoding: json